Skip to content

fix: reclaim orphaned Hetzner servers Karpenter has lost track of - #53

Draft
pkieszcz wants to merge 4 commits into
paperclipinc:mainfrom
pkieszcz:fix/orphaned-server-recovery
Draft

fix: reclaim orphaned Hetzner servers Karpenter has lost track of#53
pkieszcz wants to merge 4 commits into
paperclipinc:mainfrom
pkieszcz:fix/orphaned-server-recovery

Conversation

@pkieszcz

@pkieszcz pkieszcz commented Aug 24, 2026

Copy link
Copy Markdown
Contributor

A Hetzner server can outlive Karpenter's record of it. If the operator dies between the hcloud create call and writing the provider ID to the NodeClaim — a lost leader election, an evicted pod, an API-server timeout — the machine boots and runs with nothing pointing at it.

Every retry then collides on the server name (uniqueness_error), so the launch never succeeds, Karpenter eventually deletes the NodeClaim, and the server bills indefinitely with no owner.

Karpenter core does not reclaim it. nodeclaim/garbagecollection deletes NodeClaims whose instance has disappeared, never the reverse — that direction is the provider's responsibility, and both the AWS and Azure providers ship their own instance GC for it.

The shape of the failure, from a real occurrence:

T+0s    hcloud Create returns; the servers exist
T+27s   Failed to renew lease: context deadline exceeded
T+33s   panic: leader election lost
        karpenter/pkg/operator/operator.go
T+1m50s uniqueness_error, repeating
T+5m    deleted nodeclaim
        ... servers still running, no NodeClaim, nothing will reclaim them

No amount of in-process error handling closes that window: the process is gone between the API call and the write.

Two mechanisms

Adoption — on uniqueness_error, take the existing server over instead of retrying into the same collision. After a crash the server's own name and labels are the only surviving record, which is exactly what this reads.

It is narrow by design: the server must carry this cluster's and this NodeClaim's labels, be running or still starting, and match the requested type, location and image. An unresolved location fails closed. Relatedly, a create whose follow-up actions fail now deletes the server it just made — on a context detached from the caller's, since WaitFor returns ctx.Err() on cancellation and reusing that context would fail the cleanup before it issued a request.

A sweep every two minutes reclaims servers Karpenter no longer claims, along with the Node objects they left behind.

The sweep deletes machines — what bounds it

  • Only servers carrying this installation's ownership labels, re-checked in the controller rather than trusting the provider's list filter.
  • Ownership also reads the karpenter.sh/nodeclaim label, not just the NodeClaim provider ID, so an in-flight create whose ID was never recorded is still recognised as owned.
  • Never a node that is registered and still Ready. Registration is read from karpenter.sh/registered, which Karpenter stamps itself — deliberately not the unregistered taint, which only reaches a node if the NodePool declares it as a startupTaint, so a cluster that does not configure it would have every orphan look registered.
  • A server must be seen unowned on several consecutive sweeps, and every path that declines to act resets the count.
  • Provider IDs mapping to more than one Node are skipped rather than resolved by guessing; Node deletes carry a UID precondition.
  • List failures and undeletable servers are logged, counted and stepped over — returning an error drops RequeueAfter and ratchets controller-runtime's backoff toward its cap, delaying every other orphan.

Adopting this safely

INSTANCE_GARBAGE_COLLECTION_MODE=observe runs every check and reports what it would reclaim, changing nothing — not the fleet, not the servers' labels, nothing outside the cluster. On an existing fleet that is the way in: watch karpenter_hetzner_orphaned_server_gc_total{result="would_reap"} and the WouldGarbageCollect events, then switch to enabled.

Neither AWS nor Azure offers this. They also default to on, which they have earned with years of fleet-hours that this has not.

Checked against the AWS and Azure providers

AWS Azure here
Node deleted after the instance delete succeeds same same
Ready / registration guard none none yes
Grace source instance CreationTimestamp instance CreationTimestamp consecutive sweeps, in-process
Grace length 30s 5 min ~6 min
Own cloudProvider.List() yes, 10s→2min yes, 2min yes, 2min

Deleting the Node after the instance, and running an independent List on its own cadence, both match. The Ready guard is a deliberate divergence: a NodeClaim loss that is not per-instance — a CRD reinstall, an etcd restore — makes every server unclaimed at once, and without the guard the sweep becomes a fleet-wide termination with no eviction, drain or volume detachment. Both references look exposed to that. Happy to drop it and match them if you would rather not carry the difference.

On grace: both references derive it from the instance's creation timestamp, which is durable but measures machine age — a server owned for hours loses all protection the moment its NodeClaim blips. Counting consecutive unowned sweeps measures the right quantity but resets when the process does, so the sweep additionally requires that this process has been watching for a full window before it may reclaim anything. Instability delays reclamation rather than skipping it forever or authorising it on a short history.

An earlier revision of this branch stored that clock on the Hetzner server as a label. It was withdrawn: it made a safety-critical invariant depend on a remote write that can fail silently, on a clock written by a process that may be gone, and on state anyone can edit with one CLI command. Core solves the same problem in memory (nodeclaim/consistency) and reserves durable timeouts for status conditions on objects it owns.

Two caveats

CLUSTER_NAME is not guaranteed unique, and it previously decided what the provider would touch. Now that unclaimed servers are deleted, two clusters sharing a name in one Hetzner project would delete each other's fleets. Ownership therefore also stamps the kube-system namespace UID. A missing UID is treated as ours — servers predating this carry none, and refusing them would strand every pre-existing orphan — so until a fleet has rolled, distinct names remain the thing to get right.

A rebuilt control plane mints a new UID, so the previous incarnation's servers are refused rather than reclaimed. result="skipped_foreign_cluster" is the signal, the log names both possible causes, and the README documents relabelling as the recovery.

Verification

Unit tests throughout, RED-first, with each guard mutation-tested — reverting it makes exactly its test fail. Beyond that the sweep has been exercised against a real fleet: it reclaimed precisely the orphaned servers, left every healthy worker and every control-plane node untouched, and on clusters with no orphans ran and correctly did nothing.

No RBAC change beyond events.k8s.io, which the manager's event recorder requires; nodes delete and namespaces get were already granted. That one-line grant is the same change as #52 — happy to drop the hunk and depend on it if that lands first.

Known gaps

  • Most orphans never registered a Node, so the events supplement the log and metric rather than replacing them.
  • The sweep's List duplicates core's on the same cadence — as it does on AWS and Azure.
  • Grace is per-process. The startup window makes that safe, but it does mean a frequently-restarting operator reclaims more slowly.

Split from a companion change that reports nodes Karpenter cannot price; unrelated failure, read-only, so it did not belong behind a controller that deletes machines.

@pkieszcz pkieszcz changed the title fix: recover and reclaim orphaned servers fix: reclaim orphaned servers and report nodes Karpenter cannot price Aug 24, 2026
@pkieszcz
pkieszcz force-pushed the fix/orphaned-server-recovery branch from 3e3143c to d194bef Compare August 25, 2026 06:49
@pkieszcz pkieszcz changed the title fix: reclaim orphaned servers and report nodes Karpenter cannot price fix: reclaim orphaned Hetzner servers Karpenter has lost track of Aug 25, 2026
When the controller dies between the Hetzner create call and persisting the
provider ID, the server keeps running with no owner. Every retry then collides
on the server name, Karpenter eventually deletes the NodeClaim, and the server
bills indefinitely. A lost leader election was enough to strand several
servers this way, some of them running for days before anyone noticed.

That crash window cannot be closed by error handling, so recover from it
instead: on uniqueness_error, look the server up by name and adopt it.

Adoption is deliberately narrow. Requiring this cluster's and this NodeClaim's
labels is what makes it safe to assume the machine was built from these inputs,
since userData, SSH keys and the placement group are not verifiable afterwards.
Server type, location and image are checked directly, because Karpenter derives
the NodeClaim's capacity, zone and image from the offering selected on THIS
attempt: a mismatch would advertise capacity the machine lacks, or a zone its
volumes cannot reach. An unresolved location fails closed rather than skipping
the check, as that is the case where the zone is most likely wrong. A server
that is not running or still starting is refused, since adoption cannot wait on
create actions the way the normal path does.

A create whose actions fail now deletes the server it just made rather than
abandoning it, which keeps adoption's only input the crash case, where the
machine is healthy. That cleanup runs on a context detached from the caller's:
WaitFor returns ctx.Err() when the caller's context is cancelled, which is the
likeliest way to reach the branch at all, so reusing it would fail the delete
before it issued a request and leak the very server being reclaimed.

Adoption outcomes are counted separately from creates -- including declines and
lookup failures, since a NodeClaim retrying into a collision adoption keeps
refusing is otherwise invisible while the machine bills.
Karpenter core's garbage collector runs in one direction only: it deletes
NodeClaims that have no instance. Nothing terminates an instance that has no
NodeClaim, so that direction is the cloud provider's responsibility. Without it
an orphaned server runs and bills forever.

Sweep every two minutes over the servers this installation created, terminating
those Karpenter no longer claims along with the Node object left behind.
Ownership is read from the karpenter.sh/nodeclaim label as well as the NodeClaim
provider ID, so a server whose ID was never recorded is still recognised while
its Create is in flight, and the ownership labels are re-checked here rather
than trusting the caller's list filter.

Grace is counted in consecutive sweeps that found nothing standing in the way of
reclaiming the server -- not sweeps since its NodeClaim went missing, and not
elapsed time. Machine age is the wrong quantity: a server owned for hours would
lose all protection the moment its NodeClaim blipped. Counting observations also
removes any dependence on comparing Hetzner's clock to this pod's. Crucially the
count only advances on sweeps that cleared every guard, so a machine spared for
hours never sits on a spent counter where a single briefly-NotReady kubelet
would destroy it; once sparing stops it must earn a complete fresh window.

The sweep declines to act whenever the state is not clearly an orphan. A node
that is registered and still Ready is left alone, since a lost NodeClaim (a CRD
reinstall, an etcd restore) must never become a fleet-wide termination with no
eviction or drain. Registration is read from karpenter.sh/registered, which
Karpenter stamps itself and treats as the definition of registered; the
unregistered taint is deliberately not used, as it reaches a node only when the
NodePool declares it as a startupTaint. A node the CCM has not yet stamped with
a provider ID is matched by name rather than treated as absent. Provider IDs
mapping to more than one Node are skipped rather than resolved by guessing, and
Node deletes carry a UID precondition so a same-named replacement is never
removed.

Failures never cost the cadence: list errors and undeletable servers are logged,
counted, and stepped over, because returning an error drops RequeueAfter and
ratchets controller-runtime's backoff toward its cap. Sweeps that cannot
complete are recorded as sweep_failed, so a permanently broken sweep is
distinguishable from a cluster that simply has no orphans.

DISABLE_INSTANCE_GARBAGE_COLLECTION pauses the sweep for maintenance that
removes NodeClaims wholesale, without also stopping provisioning and disruption.
An unrecognised value is rejected rather than defaulted: the moment an operator
reaches for this flag is a maintenance window, and a typo failing open would
reap the fleet they were protecting.

The chart already grants node delete, which Karpenter core requires for
termination, so no RBAC change is needed.
CLUSTER_NAME is operator-supplied and nothing enforces uniqueness. Two
clusters sharing one in a single Hetzner project each read the other's servers
as their own. That was harmless while the label only scoped listings; it is not
now that unclaimed servers are deleted, so a copied values.yaml or a blue/green
rebuild could have one cluster terminate another's fleet.

Stamp the UID of the cluster's kube-system namespace on every server created.
It is unique per cluster, stable for the cluster's lifetime, needs no state of
our own, and is readable with the RBAC the chart already grants. Both the
adoption path and the sweep refuse a server whose UID is present and different.

A missing UID is treated as ours: servers created before this change carry
none, and refusing them would strand every pre-existing orphan and break
adoption for machines this cluster really did create. Distinct names therefore
remain the thing to get right for existing fleets; new servers are protected
regardless.

Declining silently would not be enough. A name collision would look exactly
like a cluster with no orphans, and nobody would learn two clusters share a
name until something else broke. The sweep reports each colliding cluster once,
naming both UIDs -- once per cluster rather than once per sweep, so it stays
readable.
Three changes aimed at making this safe to adopt on a fleet nobody has audited.

OBSERVE MODE. This controller deletes machines, and an operator adopting it
could otherwise only learn what it would do to their fleet by letting it do it.
A kill switch is reactive; observe mode is not. It runs every check, emits every
signal, and changes nothing -- not the fleet, not the servers' labels, nothing
outside the cluster. INSTANCE_GARBAGE_COLLECTION_MODE replaces the boolean,
since three states do not fit in one, and an unrecognised value refuses to start
rather than falling back to the mode that deletes.

A STARTUP GRACE. Grace is counted in consecutive sweeps, which a restart or a
leader handover resets -- and the instability that strands servers is exactly
what causes those. A fresh process could otherwise reach the threshold having
watched the cluster for only a few minutes. It must now also have been sweeping
for a full window before it may reclaim anything, so operator instability delays
reclamation rather than either skipping it forever or authorising it on a short
history.

This is what makes in-process counting safe rather than merely simple, and it
follows core: nodeclaim/consistency keeps first-seen times in an in-process
cache and writes nothing durable. Core's durable timeouts hang off status
conditions on objects it owns, never off the cloud resource. An earlier revision
of this branch stored the clock on the Hetzner server itself; it was withdrawn
because it made a safety-critical invariant depend on a remote write that can
fail silently, on a clock written by a process that may be gone, and on state an
operator can edit with one CLI command.

NODE EVENTS. Reclamations are recorded as GarbageCollected, and observe-mode
candidates as WouldGarbageCollect, typed Normal -- reclaiming an orphan is this
controller working, not a fault, and a cluster alerting on Warning events
against Nodes should not page every time the sweep does its job. Most orphans
never registered a Node, so this supplements the log and the metric rather than
replacing them, and the doc comment says so.

Three chart-level holes closed alongside them. The ClusterRole now grants
events.k8s.io, without which the manager's recorder has every event rejected 403
while the README tells operators to validate observe mode by reading them. The
mode is emitted unconditionally rather than through a "with" block that skips
falsy values, so "mode: false" -- the natural typo when migrating from the
boolean this replaces -- reaches the parser and stops the operator instead of
silently defaulting to enabled. And a "fail" guard rejects an upgrade still
carrying the removed "disabled" key, which Helm would otherwise merge while the
sweep ran on a fleet the operator believed was paused.

The clock is injected, matching core, so the startup grace is testable without
reaching into a fake provider.
@pkieszcz
pkieszcz force-pushed the fix/orphaned-server-recovery branch from d194bef to dc43935 Compare August 25, 2026 06:54
@stubbi

stubbi commented Aug 25, 2026

Copy link
Copy Markdown
Contributor

Directional answers while this is in draft, so you're not blocked on them:

  • Keep the Ready/registration guard. The divergence from AWS/Azure is justified for exactly the reason you give — a non-per-instance NodeClaim loss (CRD reinstall, etcd restore) must not become a fleet-wide unattended termination. Don't drop it to match the references.
  • observe as the default is right for the first release of a controller that deletes machines. We can revisit the default after it has fleet-hours behind it.
  • Drop the events.k8s.io RBAC hunk and rebasefix: update rbac to support modern event recorder events.k8s.io #52 is landing that grant separately.
  • The govulncheck failure is not yours: fixed on main by chore: bump Go to 1.26.6 #57 (Go 1.26.6). A rebase after it merges goes green.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants